Image displaying method, information processing apparatus, and non-transitory computer-readable storage medium storing program

ABSTRACT

An image displaying method including accepting from a user setting operation of setting a first surface of an object located in a real space as a plane onto which a projection image is virtually displayed, and displaying a first superimposed image in which a setting image showing a plane containing the first surface is superimposed on a captured image generated by capturing an image of the object.

The present application is based on, and claims priority from JPApplication Serial Number 2022-082102, filed May 19, 2022, thedisclosure of which is hereby incorporated by reference herein in itsentirety.

BACKGROUND 1. Technical Field

The present disclosure relates to an image display method, aninformation processing apparatus, and a non-transitory computer-readablestorage medium storing program.

2. Related Art

What is called an augmented reality (AR) technology has been developedto assist a user to acquire visual information by superimposing, forexample, computer graphics (CG) on a captured image as a result ofcapturing an image of a real space and then displaying the superimposedimage. For example, JP-A-2011-203824 discloses an image processingapparatus that identifies the position of an object in a real spacebased on an input image generated by capturing an image of the objectwith an imaging device and characteristics data representing thecharacteristics of the appearance of the object. The image processingapparatus determines the position of an image superimposed on the inputimage based on the identified position of the object.

When the image processing apparatus described in JP-A-2011-203824 storesno characteristics data on the object to be captured in the form of animage, the position of the object cannot be identified, resulting in aproblem of inappropriate positioning of the image to be superimposed.

SUMMARY

An image displaying method according to an aspect of the presentdisclosure includes accepting from a user setting operation of setting afirst surface of an object located in a real space as a plane onto whicha projection image is virtually displayed, and displaying a firstsuperimposed image in which a setting image showing a plane containingthe first surface is superimposed on a captured image generated bycapturing an image of the object.

An information processing apparatus according to another aspect of thepresent disclosure includes a processing device, and the processingdevice accepts from a user setting operation of setting a first surfaceof an object located in a real space as a plane onto which a projectionimage is virtually displayed, and outputs to a display device a firstsuperimposed image in which a setting image showing a plane containingthe first surface is superimposed on a captured image generated bycapturing an image of the object.

A program according to another aspect of the present disclosure causes aprocessing device to accept from a user setting operation of setting afirst surface of an object located in a real space as a plane onto whicha projection image is virtually displayed, and output to a displaydevice a first superimposed image in which a setting image showing aplane containing the first surface is superimposed on a captured imagegenerated by capturing an image of the object.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a diagrammatic view illustrating a space in which an object islocated.

FIG. 2 is a diagrammatic view for describing a first superimposed image.

FIG. 3 is a diagrammatic view for describing a second superimposedimage.

FIG. 4 is a diagrammatic view for describing another first superimposedimage.

FIG. 5 is a diagrammatic view illustrating detection operation.

FIG. 6 is a diagrammatic view illustrating setting operation.

FIG. 7 is a diagrammatic view illustrating the displayed firstsuperimposed image.

FIG. 8 is a diagrammatic view for describing a monitor image.

FIG. 9 is a diagrammatic view for describing a notification image.

FIG. 10 is a block diagram showing the configuration of a smartphoneaccording to an embodiment.

FIG. 11 is a block diagram showing the configuration of a storage deviceaccording to the embodiment.

FIG. 12 is a flowchart for describing the action of the smartphoneaccording to the embodiment.

FIG. 13 is a diagrammatic view for describing another secondsuperimposed image.

FIG. 14 is a diagrammatic view for describing a still another firstsuperimposed image.

DESCRIPTION OF EXEMPLARY EMBODIMENTS

A preferable embodiment according to the present disclosure will bedescribed below with reference to the accompanying drawings. In thedrawings, the dimensions and scale of each portion differ from actualvalues in some cases, and some of the portions are diagrammaticallydrawn for ease of understanding. The scope of the present disclosure isnot limited to the embodiment unless particular restrictions on thepresent disclosure are made in the following description. In the presentspecification and the claims, when a numerical range is expressed byusing “Φ to Ψ” (Φ and Ψ are both numerical values), the range includesthe numerical values of the upper limit (Ψ) and the lower limit (Φ). Theupper limit (Ψ) and the lower limit (Φ) are expressed in the same unit.

1. Embodiment

In the embodiment, an image displaying method, an information processingapparatus, and a program according to the present disclosure will bedescribed by illustrating a smartphone that acquires a captured image bycapturing an image of an object having a side surface set as a planeonto which a predetermined image is virtually displayed, superimposes animage showing a plane containing the side surface on the captured image,and displays the superimposed image.

1.1. Overview of Smartphone

A smartphone 1 according to the present embodiment will be describedbelow with reference to FIGS. 1 to 4 . FIG. 1 is a diagrammatic viewillustrating a real space in which an object Ob 1 is located. FIGS. 2 to4 are diagrammatic views for describing images displayed on a touchpanel 16 provided in the smartphone 1.

The object Ob1 is a box-shaped object having surfaces W2, W3, and W4.The surface W2 is one of the bottom surfaces of the object Ob1 that isthe bottom surface being not in contact with a floor surface WF. Thesurface W3 is one of the side surfaces of the object Ob1 that is theside surface parallel to a wall surface W1. The surface W4 is one of theside surfaces of the object Ob1 that is the side surface perpendicularto the wall surface W1. The object Ob1 is in contact with the wallsurface W1 via the side surface opposite from the surface W3. A plane H1is a conceptual plane containing the surface W3. That is, the plane H1does not really exist in the real space and is therefore not directlyvisible to a user U. Note that the object Ob1 does not necessarily havethe shape of a box and may have any other shape as long as the shape hasat least one flat surface.

In the present embodiment, the wall surface W1, the surface W3, and theplane H1 are parallel to Z- and X-axes. The Z-axis is an axis parallelto the vertical direction. Furthermore, the Z-axis is perpendicular tothe floor surface WF. Out of the directions parallel to the Z-axis, thedownward vertical direction is called a direction −Z, and the oppositedirection of the direction −Z is called a direction +Z. The X-axis isperpendicular to the Z-axis and parallel to the wall surface W1, thesurface W3, and the plane H1. Furthermore, the X-axis is perpendicularto the surface W4. Out of the directions parallel to the X-axis, thedirection from the surface W4 toward the side surface opposite from thesurface W4 is called a direction −X, and the opposite direction of thedirection −X is called a direction +X. An axis perpendicular to the Z-and X-axes is referred to as a “Y-axis”. The Y-axis is perpendicular tothe wall surface W1, the surface W3, and the plane H1. Out of thedirections parallel to the Y-axis, the direction from the surface W3toward the wall surface W1 is called a direction +Y, and the oppositedirection of the direction +Y is called a direction −Y.

With the smartphone 1 being in contact with the surface W3 of the objectOb1, the user U performs the operation of setting the surface W3 as aplane in which a projection image GP1, which will be described later, isvirtually displayed. Specifically, the user U performs the operationwith an imaging device 18 provided in the smartphone 1 being in contactwith the surface W3. In the present embodiment, “virtually displayed”means that the smartphone 1 displays an image captured with the imagingdevice 18 of the smartphone 1 through capturing an image of the realspace with another image superimposed on the real space image. In thepresent embodiment, the operation of setting the surface W3 as the planein which the projection image GP1 is virtually displayed may be referredto as “setting operation”. The smartphone 1 accepts the settingoperation from the user U with the smartphone 1 located at the surfaceW3 of the object Ob1. The setting operation may, for example, be longpressing the touch panel 16 for a predetermined period or longer. Thesetting operation may instead be the operation performed on a virtualoperator representing the setting operation, such as a finalizing buttonimage displayed on the touch panel 16.

FIG. 2 is a diagrammatic view for describing a first superimposed imageGM11. The first superimposed image GM11 is an image displayed on thetouch panel 16 when the smartphone 1 accepts the setting operation fromthe user U with the smartphone 1 located at the surface W3 of the objectOb1. The first superimposed image GM11 is an image in which a settingimage GV1 is superimposed on a captured image GC1 generated by theimaging device 18 through capturing an image of the object Ob1. Thecaptured image GC1 contains images GOb1 and GWF. The image GOb1 containsimages GW3 and GW4. The image GW3 is an image showing the surface W3.The image GW4 is an image showing the surface W4. The image GWF is animage showing the floor surface WF. The setting image GV1 is an imageshowing the plane H1 containing the surface W3. The setting image GV1represents a virtual plane in which the projection image GP1 isvirtually displayed.

Since the plane H1 is not directly visible to the user U, a portion ofor the entire region of the setting image GV1 is preferablysemitransparent. The user U can thus compare the user U's own field ofview with the captured image GC1 contained in the first superimposedimage GM11. The setting image GV1 is an image showing the position ofthe plane in which the projection image GP1 is virtually displayed. InFIG. 2 , the image GW3, which is the image showing the surface W3,coincides with a portion of the setting image GV1. Specifically, one ofthe four edges of the image GW3 coincides with a portion of the bottomedge of the setting image GV1. The user U can thus see that the plane inwhich the projection image GP1 is virtually displayed is the surface W3by checking the setting image GV1.

After accepting the setting operation from the user U, the smartphone 1acquires the captured image GC1 by capturing an image of the object Ob1.The smartphone 1 further generates the first superimposed image GM11 bysuperimposing the setting image GV1 on the captured image GC1. Thesmartphone 1 then displays the first superimposed image GM11 on thetouch panel 16.

FIG. 3 is a diagrammatic view for describing a second superimposed imageGM21. The second superimposed image GM21 is an image in which theprojection image GP1 is superimposed on the image GW3 contained in thecaptured image GC1.

When the smartphone 1 accepts the operation of displaying the projectionimage GP1 from the user U, the smartphone 1 generates the secondsuperimposed image GM21 by superimposing the projection image GP1 on thecaptured image GC1. The smartphone 1 then displays the secondsuperimposed image GM21 on the touch panel 16. That is, the smartphone 1can display the projection image GP1 superimposed on the image GW3showing the surface W3. In other words, the smartphone 1 can virtuallydisplay the projection image GP1 on the surface W3 by using the touchpanel 16. The second superimposed image GM21 may or may not contain thesetting image GV1. The second superimposed image GM21 in the presentembodiment does not contain the setting image GV1. When the secondsuperimposed image GM21 contains the setting image GV1, the user Ureadily understand the positional relationship between the surface setby the user U's setting operation and the projection image GP1. When thesecond superimposed image GM21 does not contain the setting image GV1,the smartphone 1 can present the user U the second superimposed imageGM21 more closely showing the situation in which the image is actuallyprojected onto the surface W3. In the present embodiment, the operationof displaying the projection image GP1 may be referred to as “displayoperation”. The display operation may, for example, be the operation oftapping the setting image GV1 contained in the first superimposed imageGM11 displayed on the touch panel 16. The display operation may insteadbe the operation of tapping a virtual operator displayed on the touchpanel 16 and accepting an instruction of displaying the projection imageGP1.

FIG. 4 is a diagrammatic view for describing a first superimposed imageGM12. The first superimposed image GM12 is an image in which a settingimage GV2 is superimposed on the captured image GC1. The setting imageGV2 is an image showing a plane different from the plane H1, that is, aplane that does not contain the surface W3. For example, when the user Udoes not correctly perform the setting operation, specifically, when thesetting operation is performed at a position where the imaging device 18provided in the smartphone 1 is separate from the surface W3 in thedirection −Y, the smartphone 1 displays the first superimposed imageGM12 on the touch panel 16. That is, when the first superimposed imageGM12 containing the setting image GV2 is displayed on the touch panel16, the user U can ascertain that the setting operation has not beenperformed with the imaging device 18 located at the surface W3. On theother hand, when the first superimposed image GM11 containing thesetting image GV1 is displayed on the touch panel 16 as shown in FIG. 2, the user U can ascertain that the setting operation has been performedwith the imaging device 18 located at the surface W3.

When the setting operation has not been performed with the imagingdevice 18 located at the surface W3, the surface W3 has not been set asthe plane in which the projection image GP1 is virtually displayed. Thatis, the smartphone 1 cannot in some cases display the projection imageGP1 superimposed on the image GW3 showing the surface W3. When the firstsuperimposed image GM12 is displayed on the touch panel 16, the user Ucan superimpose the projection image GP1 on the image GW3 by performingthe setting operation again with the smartphone 1 being in contact withthe surface W3 of the object Ob1. In the present embodiment, performingthe setting operation again may be referred to as “resetting operation”.

1.2. Procedure Up to Point where First Superimposed Image is Displayed

The procedure up to the point where the first superimposed image GM11 isdisplayed on the touch panel 16 will be described with reference toFIGS. 5 to 9 .

FIG. 5 is a diagrammatic view illustrating detection operation. Thesmartphone 1 has the function of detecting a flat surface of the objectOb1 by using image processing, for example, to extract characteristicpoints from a captured image generated by capturing an image of the flatsurface of the object Ob1. When the surface of the object Ob1 isdetected, the smartphone 1 sets the surface of the object Ob1 as theplane in which the projection image GP1 is virtually displayed. That is,when the surface W3 of the object Ob1 is detected, the smartphone 1 setsthe surface W3 as the plane in which the projection image GP1 isvirtually displayed.

The smartphone 1 accepts from the user U the operation of instructingdetection of the surface W3 of the object Ob1 with the captured imagegenerated by capturing an image of the object Ob1 displayed on the touchpanel 16. When the operation of instructing detection of the surface W3of the object Ob1 is accepted, the smartphone 1 attempts to detect thesurface W3 based on the captured image. In the present embodiment, theoperation of instructing detection of the surface W3 of the object Ob1may be referred to as “detection operation”. The detection operationmay, for example, be the operation of tapping an image showing theobject Ob1 contained in the captured image displayed on the touch panel16. The detection operation may instead be the operation of tapping avirtual operator displayed on the touch panel 16 and accepting aninstruction of performing the surface detection.

FIG. 8 is a diagrammatic view for describing a monitor image GC2. Themonitor image GC2 is a captured image generated by the imaging device 18through capturing an image of the object Ob1. The monitor image GC2contains images GOb1 a and GWFa. The image GOb1 a contains an image GW3a. The image GW3 a is an image showing the surface W3. The image GWFa isan image showing the floor surface WF.

When image processing is used to extract a characteristic point from animage, a point that allows accurate determination of the positionthereof is typically extracted as the characteristic point.Specifically, a characteristic point is preferably a point where theamount of change in luminance or any other factor between adjacentpixels is large both in the vertical and horizontal directions of theimage. A location where the amount of change in luminance or any otherfactor between adjacent pixels is large both in the vertical andhorizontal directions of the image may, for example, be an angularportion contained in the image. Since the monitor image GC2 contains noangular portion as seen from FIG. 8 , no characteristic point isunlikely to be accurately extracted through image processing. Therefore,when the user U performs the detection operation with the monitor imageGC2 displayed on the touch panel 16, the smartphone 1 may not be able todetect the surface W3.

When the surface W3 is not detected, the smartphone 1 prompts the user Uto perform the setting operation. For example, the smartphone 1 maydisplay a message on the touch panel 16 to prompt the user U to performthe setting operation. FIG. 9 is a diagrammatic view for describing anotification image GN1. The notification image GN1 is an imagecontaining a message that prompts the user U to perform the settingoperation. Displaying the notification image GN1 on the touch panel 16is, in other words, outputting notification about the setting operation.

FIG. 6 is a diagrammatic view illustrating the setting operation. Theuser U sets the surface W3 as the plane in which the projection imageGP1 is virtually displayed by long-pressing the touch panel 16 for apredetermined period or longer, that is, performing the settingoperation with the imaging device 18 provided in the smartphone 1 beingin contact with the surface W3. When the smartphone 1 accepts thesetting operation from the user U, the smartphone 1 sets the surface W3as the plane in which the projection image GP1 is virtually displayed.The smartphone 1 sets the position of the imaging device 18 at the pointof time when the smartphone 1 accepts the setting operation as areference position. The smartphone 1 further sets the optical axis ofthe imaging device 18 at the point of time when the smartphone 1 acceptsthe setting operation as a reference axis. The optical axis of theimaging device 18 at the point of time when the smartphone 1 accepts thesetting operation is referred to as an “optical axis L11”. The opticalaxis L11 is perpendicular to the surface W3 and parallel to the Y-axis.The point of time when the smartphone 1 accepts the setting operation isreferred to as a “first point of time”.

FIG. 7 is a diagrammatic view illustrating the displayed firstsuperimposed image GM11. After accepting the setting operation, thesmartphone 1 controls the imaging device 18 to acquire the capturedimage GC1 by capturing an image of the object Ob1 from a positionseparate from the surface W3 in the direction −Y. The point of time whenthe imaging device 18 acquires the captured image GC1 is referred to asa “second point of time”. The displacement of the imaging device 18 thatoccurs between the first and second points of time is referred to as a“displacement DP1”. The term “displacement” used herein refers to avector representing the change in position of the imaging device 18. Theoptical axis of the imaging device 18 at the second point of time isreferred to as an “optical axis L12”. The smartphone 1 generates thesetting image GV1 based on the displacement DP1 and a change in postureof the imaging device 18 that occurs between the first and second pointsof time. The smartphone 1 further generates the first superimposed imageGM11 by superimposing the setting image GV1 on the captured image GC1.The smartphone 1 then displays the first superimposed image GM11 on thetouch panel 16.

The change in the posture of the imaging device 18 is expressed, forexample, by three-dimensional angles of rotation around the optical axisof the imaging device 18 as the rolling axis, an axis parallel to thewidthwise direction of the smartphone 1 as the pitching axis, and anaxis parallel to the lengthwise direction of the smartphone 1 as theyawing axis with the imaging device 18 being the origin. The change inthe posture of the imaging device 18 is specifically expressed by therolling, pitching, and yawing angles at the point of time when thesmartphone 1 captures an image of the object Ob1, provided that therolling, pitching, and yawing angles at the point of time when thesmartphone 1 accepts the setting operation are all 0°. That is, thechange in the posture of the imaging device 18 that occurs between thefirst and second points of time is expressed by changes in the rolling,pitching, and yawing angles that occur when the optical axis of theimaging device 18 changes from the optical axis L11 to the optical axisL12.

1.3. Configuration and Functions of Smartphone

The configuration and functions of the smartphone 1 according to thepresent embodiment will be described below with reference to FIGS. 10and 11 .

FIG. 10 is a block diagram showing the configuration of the smartphone 1according to the embodiment. The smartphone 1 includes a storage device10, which stores a variety of pieces of information, a processing device12, which controls the action of the smartphone 1, a sensor 14, whichoutputs a signal based on a change in the position of the smartphone 1and a signal based on a change in the posture of the smartphone 1, thetouch panel 16, which displays a variety of images and accepts operationfrom the user U, and the imaging device 18, which captures an image ofthe object Ob1. The processing device 12 functions as a displaycontroller 120, an imaging controller 121, an input manager 122, animage generator 123, a detector 124, and a space manger 125. The touchpanel 16 includes a display section 160 and an input section 161. Theimaging device 18 includes an imaging lens 180 and an imaging device181.

The storage device 10 includes, for example, a volatile memory, such asa RAM, and a nonvolatile memory, such as a ROM. RAM is an abbreviationfor a random access memory. ROM is an abbreviation for a read onlymemory.

FIG. 11 is a block diagram showing the configuration of the storagedevice 10 according to the embodiment. The nonvolatile memory providedin the storage device 10 stores a program 100, which defines the actionof the smartphone 1, space information 101 containing informationrepresenting the position and shape of a detected object, positioninformation 102 based on the change in the position of the imagingdevice 18, posture information 103 based on the change in the posture ofthe imaging device 18, projection image information 104 representing theprojection image GP1, imaging information 105 representing the capturedimage GC1, monitor information 106 representing the monitor image GC2,setting image information 107 representing the setting image GV1,notification information 108 representing the notification image GN1,and superimposed image information 110. The superimposed imageinformation 110 contains first superimposed image information 111representing the first superimposed image GM11 and second superimposedimage information 112 representing the second superimposed image GM21.

The volatile memory provided in the storage device 10 is also used bythe processing device 12 as a work area when the processing device 12executes the program 100.

A portion or the entirety of the storage device 10 may be provided in anexternal storage apparatus, an external server, or any other component.A portion or the entirety of the variety of pieces of information storedin the storage device 10 may be stored in the storage device 10 inadvance, or may be acquired from the external storage apparatus, theexternal server, or any other component.

The processing device 12 includes one or more CPUs. It is, however,noted that the processing device 12 may include a programmable logicdevice, such as an FPGA, in place of or in addition to the CPU. The CPUis an abbreviation for a central processing unit, and FPGA is anabbreviation for a field-programmable gate array.

The processing device 12 functions as the display controller 120, theimaging controller 121, the input manager 122, the image generator 123,the detector 124, and the space manager 125 shown in FIG. 10 by causingthe CPU or any other component provided in the processing device 12executes the program 100 and operates in accordance with the program100.

The display controller 120 controls the display section 160 provided inthe touch panel 16 to display a variety of images. In other words, thedisplay controller 120 outputs a variety of images to the displaysection 160 provided in the touch panel 16.

The imaging controller 121 controls the imaging device 18 to capture animage of the object Ob1. The imaging controller 121 then acquiresinformation representing the result of the capturing an image of theobject Ob1 from the imaging device 18. The imaging controller 121 causesthe storage device 10 to store the information acquired from the imagingdevice 18. In the present embodiment, the imaging controller 121acquires the imaging information 105 and the monitor information 106 asthe information representing the result of the capturing an image of theobject Ob1. The imaging controller 121 then causes the storage device 10to store the acquired imaging information 105 and monitor information106.

The imaging controller 121 further generates the position information102 and the posture information 103 based on the signals outputted fromthe sensor 14. In the present embodiment, the position information 102is information representing the displacement DP1. The postureinformation 103 is information representing the change in the posture ofthe imaging device 18 that occurs between the first and second points oftime. The imaging controller 121 causes the storage device 10 to storethe generated position information 102 and posture information 103.

The input manager 122 controls the input section 161 provided in thetouch panel 16 to acquire information representing the content ofoperation received from the user U. The input manager 122 furtherperforms a variety of types of determination relating to the operationaccepted from the user U.

The detector 124 detects a flat surface of the object Ob1 by using imageprocessing to extract characteristic points from the captured imagegenerated by capturing an image of the object Ob1. In other words, thedetector 124 determines whether the captured image contains an imageshowing a flat surface of the object Ob1. In the present embodiment, thedetector 124 determines whether the monitor image GC2 contains an imageshowing the surface W3 of the object Ob1. The captured image that issubject to the determination of whether the captured image contains animage showing a flat surface of the object Ob1 may be referred to as a“monitor image”.

Any known image processing technology may be used in the function ofextracting the characteristic points from the captured image. The knownimage processing technology relating to extraction of the characteristicpoints is, for example, an algorithm called “AKAZE”. No detailedtechnical description relating to the extraction of the characteristicpoints will be made in the present specification.

The space manager 125 manages the space information 101, which containsinformation representing the position and shape of an object disposed inthe real space. For example, the space manager 125 updates the spaceinformation 101 based on the position and shape of the detected surfaceof the object Ob1.

The space information 101 contains information on the position where thesmartphone 1 has accepted the setting operation. Based on the signalsoutputted from the sensor 14, the space manager 125 updates the spaceinformation 101 in such a way that the position of the smartphone 1 atthe first point of time, when the setting operation is accepted from theuser U, is the reference position. More specifically, the space manager125 updates the space information 101 in such a way that the position ofthe imaging device 18 at the first point of time is, for example, theorigin. By updating the space information 101, the space manager 125sets the plane in which the projection image GP1 is virtually displayedin such a way that the plane contains the position of the imaging device18 at the first point of time. That is, when the setting operation isaccepted from the user U with the imaging device 18 located at thesurface W3 of the object Ob1, the space manager 125 sets the surface W3as the plane in which the projection image GP1 is virtually displayed.The plane in which the projection image GP1 is virtually displayed is,for example, set so as to be perpendicular to the optical axis L11 ofthe imaging device 18 at the first point of time.

The image generator 123 generates the setting image information 107based on the space information 101, the position information 102, andthe posture information 103. In other words, the image generator 123generates the setting image GV1 based on the space information 101, theposition information 102, and the posture information 103. The imagegenerator 123 further generates the first superimposed image information111 based on the set image information 107 and the imaging information105. In other words, the image generator 123 generates the firstsuperimposed image GM11 based on the setting image GV1 and the capturedimage GC1. The image generator 123 further generates the secondsuperimposed image information 112 based on the projection imageinformation 104 and the imaging information 105. In other words, theimage generator 123 generates the second superimposed image GM21 basedon the projection image GP1 and the captured image GC1.

The sensor 14 includes an acceleration sensor and a gyro sensor. Thesensor 14 detects a change in the position of the smartphone 1 and achange in the posture of the smartphone 1. The sensor 14 outputs asignal based on the change in the position of the smartphone 1 and asignal based on the change in the posture of the smartphone 1 to theprocessing device 12.

The touch panel 16 is an apparatus including the display section 160,which displays an image, and the input section 161, which acceptsoperation inputted from the user U, with the two components integratedinto a single unit. The display section 160 is what is called a displaypanel, and includes, for example, a liquid crystal panel or an organicEL panel. EL is an abbreviation for electro-luminescence. The displaysection 160 displays a variety of images under the control of thedisplay controller 120. The input section 161 includes, for example, atransparent-sheet-shaped contact sensor. The input section 161 isprovided so as to cover the display section 160. The input section 161uses electrostatic capacity identified by the input section 161 and anobject in contact therewith to detect a touch position and outputs datarepresenting the detected touch position to the processing device 12.

The imaging device 18 is, for example, a camera including the imaginglens 180, which collects light, and the imaging device 181, whichconverts collected light into an electric signal. The optical axes L11and L12 are each the optical axis of the imaging lens 180. The imagingdevice 181 is, for example, an image sensor, such as a CCD or a CMOSdevice. CCD is an abbreviation for a charge coupled device, and CMOS isan abbreviation for complementary metal oxide semiconductor. The imagingdevice 18 acquires a captured image under the control of the imagingcontroller 121. The imaging device 18 outputs information representingthe acquired captured image to the processing device 12. For example,the imaging device 18 acquires the captured image GC1 and the monitorimage GC2. The imaging device 18 further outputs the imaging information105 representing the captured image GC1 and the monitor information 106representing the monitored image GC2 to the processing device 12.

1.4. Action of Smartphone

FIG. 12 is a flowchart for describing the action of the smartphone 1according to the embodiment. The series of actions shown in theflowchart is initiated, for example, when the smartphone 1 is powered onand the touch panel 16 accepts operation inputted from the user U andrelating to the start of the action.

In step S101, the imaging controller 121 controls the imaging device 18to capture an image of the object Ob1. The imaging controller 121 thenacquires the monitor information 106 representing the result of thecapturing an image of the object Ob1 from the imaging device 18. Theimaging controller 121 causes the storage device 10 to store theacquired monitor information 106.

In step S102, the display controller 120 controls the display section160 provided in the touch panel 16 to display the monitor image GC2indicated by the monitor information 106.

In step S103, the input manager 122 determines whether the input section161 has accepted the detection operation from the user U. When the inputsection 161 has accepted the detection operation from the user U, thatis, when the result of the determination in step S103 is YES, the inputmanager 122 advances the process to the process in step S104. When theinput section 161 has not accepted the detection operation from the userU, that is, when the result of the determination in step S103 is NO, theinput manager 122 advances the process to the process in step S101.

When the input section 161 has not accepted the detection operation fromthe user U, the smartphone 1 causes the imaging device 18 to keepcapturing images of the object Ob1 until the input section 161 acceptsthe detection operation from the user U. The smartphone 1 then keepsupdating the monitor image displayed on the touch panel 16. It isassumed in the present embodiment that the user U performs the detectionoperation at the point of time when the monitor image GC2 is displayedon the touch panel 16.

In step S104, the detector 124 determines whether the monitor image GC2contains an image showing the surface W3 of the object Ob1. When thedetector 124 determines that the monitor image GC2 contains an imageshowing the surface W3 of the object Ob1, that is, when the result ofthe determination in step S104 is YES, the detector 124 advances theprocess to the process in step S107. When the detector 124 determinesthat the monitor image GC2 contains no image showing the surface W3 ofthe object Ob1, that is, when the result of the determination in stepS104 is NO, the detector 124 advances the process to the process in stepS105.

The determination in step S104 is performed after the detector 124extracts a plurality of characteristic points from the monitor image andbased on the number, positions, and other factors of the extractedplurality of characteristic points. Therefore, when characteristicpoints cannot be correctly extracted from the monitor image, and evenwhen the monitor image contains an image showing a flat surface, thedetector 124 may determine that the monitor image contains no imageshowing a flat surface.

In step S105, the display controller 120 controls the display section160 provided in the touch panel 16 to display the notification image GN1indicated by the notification information 108.

In step S106, the input manager 122 determines whether the input section161 has accepted the setting operation from the user U. When the inputsection 161 has accepted the setting operation from the user U, that is,when the result of the determination in step S106 is YES, the inputmanager 122 advances the process to the process in step S107. When theinput section 161 has not accepted the setting operation from the userU, that is, when the result of the determination in step S106 is NO, theinput manager 122 carries out the process in step S106 again.

When the input section 161 has not accepted the setting operation fromthe user U, the input manager 122 repeats the determination in step S106until the input section 161 accepts the setting operation from the userU. The touch panel 16 preferably keeps displaying the notification imageGN1 until the input section 161 accepts the setting operation from theuser U. The user U can thus grasp the action to be taken by the user Uby checking the message contained in the notification image GN1.

In step S107, by updating the space information 101, the space manager125 sets the surface W3 as the plane in which the projection image GP1is virtually displayed.

In step S108, the imaging controller 121 controls the imaging device 18to capture an image of the object Ob1.

The imaging controller 121 then acquires the imaging information 105representing the result of the capturing an image of the object Ob1 fromthe imaging device 18. The imaging controller 121 causes the storagedevice 10 to store the acquired imaging information 105.

In step S109, the image generator 123 generates the setting imageinformation 107 representing the setting image GV1 based on the spaceinformation 101, the position information 102, and the postureinformation 103. The image generator 123 further generates the firstsuperimposed image information 111 representing the first superimposedimage GM11 based on the setting image information 107 and the imaginginformation 105.

In step S110, the display controller 120 controls the display section160 provided in the touch panel 16 to display the first superimposedimage GM11 indicated by the first superimposed image information 111.

In step S111, the input manager 122 determines whether the input section161 has accepted the resetting operation from the user U. When the inputsection 161 has accepted the resetting operation from the user U, thatis, when the result of the determination in step S111 is YES, the inputmanager 122 advances the process to the process in step S107. When theinput section 161 has not accepted the resetting operation from the userU, that is, when the result of the determination in step S111 is NO, theinput manager 122 advances the process to the process in step S112.

If the smartphone 1 is separate from the surface W3 of the object Ob1 inthe setting operation in step S106, the processes up to step S110 arecarried out so that the first superimposed image GM12 is displayed. Whenthe image displayed in step S110 is the first superimposed image GM12,the user U can grasp that the plane set as the plane in which theprojection image GP1 is virtually displayed is not the surface W3. Inthis case, the user U can perform the resetting operation with thesmartphone 1 being in contact with the surface W3 of the object Ob1 toset the surface W3 as the plane in which the projection image GP1 isvirtually displayed.

In step S112, the input manager 122 determines whether the input section161 has accepted the display operation from the user U. When the inputsection 161 has accepted the display operation from the user U, that is,when the result of the determination in step S112 is YES, the inputmanager 122 advances the process to the process in step S113. When theinput section 161 has not accepted the display operation from the userU, that is, when the result of the determination in step S112 is NO, theinput manager 122 advances the process to the process in step S108.

When the input section 161 has not accepted the display operation fromthe user U, the smartphone 1 causes the imaging device 18 to keepcapturing images of the object Ob1 until the input section 161 acceptsthe display operation from the user U. The smartphone 1 then keepsupdating the image displayed on the touch panel 16. It is assumed in thepresent embodiment that the user U performs the display operation at thepoint of time when the first superimposed image GM11 is displayed on thetouch panel 16.

In step S113, the image generator 123 generates the second superimposedimage information 112 representing the second superimposed image GM21based on the projection image information 104 and the imaginginformation 105.

In step S114, the display controller 120 controls the display section160 provided in the touch panel 16 to display the second superimposedimage GM21 indicated by the second superimposed image information 112.

When the user U performs the setting operation in step S107, theprojection image GP1 is displayed with respect, for example, to theposition of the image GW3 corresponding to the location where theimaging device 18 is located on the surface W3 at the point of time whenthe smartphone 1 accepts the setting operation. When the detector 124detects the surface W3 in step S104, the smartphone 1, for example, setsthe projection image GP1 to be virtually displayed on a portion of thesurface W3 or the entire surface W3 based on the position and shape ofthe detected surface W3.

After the process in step S114 is carried out, the processing device 12terminates the series of actions shown in the flowchart of FIG. 12 .

As described above, according to the embodiment, the smartphone 1 canset the surface W3 as the plane in which the projection image GP1 isvirtually displayed by accepting the setting operation from the user Uwith the smartphone 1 located at the surface W3 of the object Ob1. Thatis, the smartphone 1 can display the projection image GP1 superimposedon the image GW3 showing the surface W3 even when the surface W3 cannotbe detected.

According to the embodiment, the smartphone 1 can indicate the positionof the plane in which the projection image GP1 is virtually displayed bydisplaying the first superimposed image GM11 containing the settingimage GV1. That is, the user U can check whether the setting operationhas been performed with the imaging device 18 located at the surface W3.

As described above, the image displaying method according to theembodiment includes accepting from the user U the setting operation ofsetting the surface W3 of the object Ob1 located in the real space asthe plane in which the projection image GP1 is virtually displayed, anddisplaying the first superimposed image GM11, in which the setting imageGV1 showing the plane H1 containing the surface W3 is superimposed onthe captured image GC1 generated by capturing an image of the objectOb1.

The smartphone 1 according to the embodiment includes the processingdevice 12, and the processing device 12 accepts from the user U thesetting operation of setting the surface W3 of the object Ob1 located inthe real space as the plane in which the projection image GP1 isvirtually displayed, and outputs to the touch panel 16 the firstsuperimposed image GM11, in which the setting image GV1 showing theplane H1 containing the surface W3 is superimposed on the captured imageGC1 generated by capturing an image of the object Ob1.

The program 100 according to the embodiment causes the processing device12 to accept from the user U the setting operation of setting thesurface W3 of the object Ob1 located in the real space as the plane inwhich the projection image GP1 is virtually displayed, and output to thetouch panel 16 the first superimposed image GM11, in which the settingimage GV1 showing the plane H1 containing the surface W3 is superimposedon the captured image GC1 generated by capturing an image of the objectOb1.

That is, the image displaying method, the smartphone 1, and the program100 according to the present embodiment can set the plane in which theprojection image GP1 is virtually displayed, in other words, theposition on the captured image GC1 on which the projection image GP1 issuperimposed, based on the user U's operation. The smartphone 1 cantherefore display the projection image GP1 at a position specified bythe user U without using characteristic data representing thecharacteristics of the appearance of the object Ob1, which is thesubject, so that the projection image GP1 can be displayed at anappropriate position.

Furthermore, the smartphone 1 according to the present embodiment canindicate the position of the plane in which the projection image GP1 isvirtually displayed by displaying the first superimposed image GM11containing the setting image GV1. The user U can thus check whether thesetting operation has been performed as intended by the user U beforethe projection image GP1 is displayed.

In the present embodiment, the smartphone 1 is an example of the“information processing apparatus”, the program 100 is an example of the“program”, the object Ob1 is an example of the “object”, the surface W3is an example of the “first surface”, the projection image GP1 is anexample of the “projection image”, the user U is an example of the“user”, the captured image GC1 is an example of the “captured image”,the plane H1 is an example of the “plane containing the first surface”,the setting image GV1 is an example of the “setting image”, the firstsuperimposed image GM11 is an example of the “first superimposed image”,the processing device 12 is an example of the “processing device”, andthe touch panel 16 is an example of the “display device”.

In the image displaying method according to the embodiment, the capturedimage GC1 is acquired by the imaging device 18, the setting operation isperformed with the imaging device 18 located at the surface W3 of theobject Ob1, the method further includes acquiring the positioninformation 102 representing the displacement DP1 of the imaging device18 at the second point of time when the imaging device 18 acquires thecaptured image GC1 with respect to the position of the imaging device 18at the first point of time when the setting operation is accepted, andthe posture information 103 representing the change in the posture ofthe imaging device 18 at the second point of time with respect to theposture of the imaging device 18 at the first point of time, and thesetting image GV1 is generated based on the position information 102 andthe posture information 103.

That is, the smartphone 1 accepts the setting operation from the user Uwith the imaging device 18 located at the surface W3 of the object Ob1.The user U can thus accurately set the plane in which the projectionimage GP1 is virtually displayed through simple operation.

The first superimposed image GM11 is generated by superimposing thesetting image GV1, which is generated based on the position information102 and the posture information 103, on the captured image GC1. That is,for example, even when a plane that makes it difficult to extractcharacteristic points through image processing is set as the plane inwhich the projection image GP1 is virtually displayed, the smartphone 1can identify the position of the plane. The smartphone 1 can thusproperly superimpose the setting image GV1 at a predetermined position.

In the present embodiment, the imaging device 18 is an example of the“imaging device”, the position information 102 is an example of the“position information,” the posture information 103 is an example of the“posture information,” and the displacement DP1 is an example of the“displacement”.

The image displaying method according to the embodiment further includesaccepting the detection operation of instructing detection of thesurface W3 from the user U, and outputting, when the detection operationis accepted but the surface W3 is not detected, notification regardingthe setting operation.

That is, the user U can use a plurality of methods to set the plane inwhich the projection image GP1 is virtually displayed. Usability at thesetting of the plane in which the projection image GP1 is virtuallydisplayed is thus improved.

In the present embodiment, “outputting notification regarding thesetting operation” is, for example, displaying the notification imageGN1 on the touch panel 16.

The image displaying method according to the embodiment further includesdetermining, when the detection operation is accepted, whether themonitor image GC2 generated by capturing an image of the object Ob1contains an image showing the surface W3, and outputting thenotification includes outputting the notification when the result of thedetermination shows that the monitor image GC2 contains no image showingthe surface W3.

The user U can thus perform the detection operation while checking viathe monitor image a plane that the user U desires to set as the plane inwhich the projection image GP1 is virtually displayed.

In the present embodiment, the monitor image GC2 is an example of the“monitor image”. An “image showing the first surface” is an example ofthe images GW3 and GW3 a.

The image displaying method according to the embodiment further includesaccepting the display operation of displaying the projection image GP1from the user U, and displaying, when the display operation is accepted,the second superimposed image GM21, in which the projection image GP1 issuperimposed on a portion or the entirety of the region of the image GW3contained in the captured image GC1.

When the display operation is accepted from the user U, the smartphone 1superimposes the projection image GP1 on the captured image GC1. Theuser U can thus check through the second superimposed image GM21 how theprojection image GP1 looks when projected in the real space.

In the present embodiment, the second superimposed image GM21 is anexample of the “second superimposed image”.

2. Variations

The embodiment described above can be changed in a variety of manners.Specific aspects of the changes will be presented below by way ofexample. Two or more aspects arbitrarily selected from those presentedbelow by way of example may be combined with each other as appropriateto the extent that the selected aspects do not contradict each other. Inthe variations presented below by way of example, an element providingthe same effect and having the same function as the element in theembodiment described above has the same reference character used in theabove description, and no detailed description of the same element willbe made as appropriate.

2.1. Variation 1

The aforementioned embodiment has been described with reference to thecase where a projection image is virtually displayed on a surface of anobject, and a virtual projector that projects a projection image may bedisplayed along with the projection image.

FIG. 13 is a diagrammatic view for describing a second superimposedimage GM23. The second superimposed image GM23 is displayed on the touchpanel 16 of the smartphone 1. The second superimposed image GM23 is animage in which a projection image GP1 b and a projection apparatus imageGS1 are superimposed on a captured image GC3 generated by the imagingdevice 18 through capturing an image of the object Ob1. The capturedimage GC3 contains an image GOb1 b and an image GWFb. The image GOb1 bcontains images GW3 b and GW4 b. The image GW3 b is an image showing thesurface W3. The image GW4 b is an image showing the surface W4. Theimage GWFb is an image showing the floor surface WF. The projectionapparatus image GS1 is an image showing a virtual projector virtuallydisposed in the real space. The projection image GP1 b is an imagevirtually projected onto the surface W3 from the virtual projector shownby the projection apparatus image GS1. That is, the projection image GP1b is superimposed on the image GW3 b showing the surface W3. Theconfiguration described above allows, for example, a simulation of how aprojection image is displayed when a projector is disposed in the realspace.

As described above, in the image displaying method according toVariation 1, the projection image GP1 b is an image virtually projectedfrom the virtual projector virtually disposed in the real space.

That is, the smartphone 1 can display the projection image GP1 b, whichis virtually displayed on a surface of the object Ob1 disposed in thereal space, and the virtual projector, which virtually projects theprojection image GP1 b. The user U of the smartphone 1 can perform, forexample, a simulation on how a projection image is displayed when aprojector is disposed in the real space.

In Variation 1, the projection image GP1 b is an example of the“projection image”.

2.2. Variation 2

The aforementioned embodiment and variation have been described withreference to the case where a projection image is superimposed on animage showing the surface W3, in other words, the projection image isvirtually displayed on the surface W3. However, the plane in which theprojection image is virtually displayed may be a plane different fromthe surface W3. That is, the smartphone 1 may accept the settingoperation from the user U with the imaging device 18 located at asurface different from the surface W3. For example, the smartphone 1 mayaccept the setting operation from the user U with the imaging device 18located at the surface W2 of the object Ob1.

FIG. 14 is a diagrammatic view for describing a first superimposed imageGM13. The first superimposed image GM13 is an image displayed on thetouch panel 16 when the smartphone 1 accepts the setting operation fromthe user U with the smartphone 1 located at the surface W2 of the objectOb1. The first superimposed image GM13 is an image in which a settingimage GV4 is superimposed on the captured image GC1. The setting imageGV4 is an image showing a plane containing the surface W2. The settingimage GV4 is an image showing the position of the plane in which theprojection image is virtually displayed, as the setting image GV1 is.The user U can see that the plane in which the projection image isvirtually displayed is the surface W2 by checking the setting image GV4.Since the captured image GC1 contains no image showing the surface W2,the user U preferably cause the smartphone 1 to acquire a captured imagecontaining an image showing the surface W2 before performing the displayoperation. The object having a plane in which the projection image isvirtually displayed may, for example, be the wall surface W1 itself. Inthis case, the smartphone 1 accepts the setting operation from the userU with the imaging device 18 located at the wall surface W1.

2.3. Variation 3

The aforementioned embodiment and variations have been described withreference to the case where the smartphone 1 is presented by way ofexample as the information processing apparatus according to the presentdisclosure, and the smartphone 1 may instead be a tablet terminal, alaptop computer, or any other apparatus having the same functions asthose of the smartphone 1.

2.4. Variation 4

The aforementioned embodiment and variations have been described withreference to the case where the notification image GN1 containing amessage that prompts the user U to perform the setting operation whenthe surface W3 is not detected, but the present disclosure is notlimited to the aspect described above. For example, instead ofdisplaying a message, the user U may be prompted by voice to perform thesetting operation. That is, voice may be used to output notificationabout the setting operation.

What is claimed is:
 1. An image displaying method comprising: acceptingfrom a user setting operation of setting a first surface of an objectlocated in a real space as a plane onto which a projection image isvirtually displayed; and displaying a first superimposed image in whicha setting image showing a plane containing the first surface issuperimposed on a captured image generated by capturing an image of theobject.
 2. The image displaying method according to claim 1, wherein thecaptured image is acquired by an imaging device, the setting operationis performed with the imaging device located at the first surface of theobject, the setting operation is accepted at a first point of time, theimaging device acquires the captured image at a second point of time,the method further comprising acquiring position informationrepresenting displacement of the imaging device at the second point oftime with respect to a position of the imaging device at the first pointof time, and posture information representing a change in a posture ofthe imaging device at the second point of time with respect to theposture of the imaging device at the first point of time, and thesetting image is generated based on the position information and theposture information.
 3. The image displaying method according to claim1, further comprising: accepting detection operation of instructingdetection of the first surface from the user; and outputting, when thedetection operation is accepted but the first surface is not detected,notification regarding the setting operation.
 4. The image displayingmethod according to claim 3, further comprising determining, when thedetection operation is accepted, whether a monitor image generated bycapturing an image of the object contains an image showing the firstsurface, and the outputting the notification includes outputting thenotification when a result of the determining shows that the monitorimage does not contain the image showing the first surface.
 5. The imagedisplaying method according to claim 1, further comprising: acceptingdisplay operation of displaying the projection image from the user; anddisplaying, when the display operation is accepted, a secondsuperimposed image in which the projection image is superimposed on aportion or entirety of a region of an image showing the first surfaceand contained in the captured image.
 6. The image displaying methodaccording to claim 5, wherein the projection image is an image virtuallyprojected from a virtual projector virtually disposed in the real space.7. An information processing apparatus comprising A processing deviceprogrammed to accept from a user setting operation of setting a firstsurface of an object located in a real space as a plane onto which aprojection image is virtually displayed, and output to a display devicea first superimposed image in which a setting image showing a planecontaining the first surface is superimposed on a captured imagegenerated by capturing an image of the object.
 8. A non-transitorycomputer readable storage medium storing program that causes aprocessing device to accept from a user setting operation of setting afirst surface of an object located in a real space as a plane onto whicha projection image is virtually displayed, and output to a displaydevice a first superimposed image in which a setting image showing aplane containing the first surface is superimposed on a captured imagegenerated by capturing an image of the object.